Methods for operator control unit and payload communication

ABSTRACT

A method of communication between a payload and a platform, including providing power to at least one payload via a platform, periodically sending an identification message from the at least one payload to the platform, decoding the identification message with the platform and sending a response message to the at least one payload, requesting access to one or more ethernet busses supplied by the platform, and enabling the one or more ethernet busses.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S Provisional Patent ApplicationSer. No. 62/864,551, filed Jun. 21, 2019, the contents of which ishereby incorporated by references in its entirety as if fully set forthherein.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to mobile robots. Moreparticularly, the present disclosure relates to devices, systems andmethods for communication between an operator control unit and one ormore payloads.

BACKGROUND OF THE DISCLOSURE

Robots are useful in a variety of civilian, military, and lawenforcement applications. For instance, some robots may inspect orsearch buildings with structural damage caused by earthquakes, floods,or hurricanes, or inspect buildings or outdoor sites contaminated withradiation, biological agents such as viruses or bacteria, or chemicalspills. Some robots carry appropriate sensor systems for inspection orsearch tasks. Robots designed for military applications may performoperations that are deemed too dangerous for soldiers. Law enforcementapplications include reconnaissance, surveillance, bomb disposal andsecurity patrols.

Small robots are useful for many applications, and the application maybe chosen by equipping the robot with one or more accessories, oftencalled payloads. Such payload may include arms, cameras, sensors,grippers and the like. Unfortunately, common robots are limited byvarious constrains. In some cases, a robot may require a change in itspayloads. In such cases, a fielded robot with a first payload isrecalled to a maintenance depot, and its operator control unit (OCU) isupdated prior to attaching a different payload. This limitation iscumbersome, time-consuming and expensive.

For example, in military applications, unmanned systems tend to follow asimilar acquisition strategy: requirement for the system is established,OCU and unmanned systems are designed, built, tested, and fielded. Butshortly after fielding, the operating environment changes, the threatsreact, and eventually the unmanned system needs an upgrade. This poses achallenge to DoD Program Managers and the DoD technology labs throughoutthe different services. The labs are tasked to develop new payloads,which entails changing the existing software of fielded systems. In manycases, an effort to field even the simplest improvement can take yearsand the logistics costs of retrofitting and upgrading the platform candwarf the payload costs. These costs and lead times along with systemavailability make it difficult to field new capabilities to thewarfighters.

SUMMARY OF THE DISCLOSURE

In some embodiments, method of communication between at least onepayload and a platform includes providing power to the at least onepayload via a platform, periodically sending an identification messagefrom the at least one payload to the platform, decoding theidentification message with the platform and sending a response messageto the at least one payload, requesting access to one or more ethernetbusses supplied by the platform, and enabling the one or more ethernetbusses.

In some embodiments, a method of communicating between a first payloadand a second payload includes broadcasting, from the first payload, adiscovery message, receiving the discovery message with the secondpayload, responding to the broadcast message by sending, from the secondpayload, at least one definition, generating a combined model of the atleast one definition from both the first payload and the second payload,avoiding operational conflicts with other payloads, and sending updateddefinitions as required.

BRIEF DESCRIPTION OF THE DISCLOSURE

Various embodiments of the presently disclosed devices, systems, andmethods are shown herein with reference to the drawings, wherein:

FIG. 1A is a diagram of a robotic system overview;

FIG. 1B is a diagram showing an OCU in communication with multiplesplatforms;

FIG. 2 is a diagram showing a method of payload-platform initialization;

FIG. 3 is a diagram showing a method of payload discovery andinitialization;

FIG. 4 is a diagram showing a custom message object;

FIG. 5 is a diagram showing custom message encoding and decoding;

FIG. 6 is a diagram showing payload defined human-machine interface;

FIG. 7 is a diagram showing payload defined on-screen controls;

FIG. 8 is a diagram showing OCU defined capabilities and limitations;

FIG. 9 is a diagram showing payload to payload communication;

FIGS. 10A-B are schematics showing the OCU screen before and after apayload is plugged in;

FIG. 11 is one example of an unmanned platform implementing the systemand methods described herein; and

FIGS. 12A-B are schematics showings a controller being mapped to certainfunctions.

Various embodiments of the present invention will now be described withreference to the appended drawings. It is to be appreciated that thesedrawings depict only some embodiments of the invention and are thereforenot to be considered limiting of its scope.

DETAILED DESCRIPTION

Despite the various improvements that have been made to robots and theircommunication with payloads, conventional systems and methods sufferfrom some shortcomings as described above.

The present disclosure provides systems and methods designed to exercisestandard message protocols in an architecture designed for trueplug-and-play payloads. According to the present disclosure, unmannedsystems may receive swappable payloads without having to recall theOperator Control Unit (OCU) to be reprogrammed to support a new payload.

The present disclosure provides a software layer sitting on top of anopen architecture unmanned platform, implemented on both the OCU and thepayload as will be described in more detail below. The methods andsystems described herein may be implemented on different OCUs andpayloads. On an OCU, the system serves as the primary software betweenthe operator and their system. Inside a payload, the system providesinformation to the rest of the system over standard protocols.

The OCU is the key interface between the human operator and the unmannedsystem providing safety standoff or remote surveillance. An OCU mayinclude one or more computer processors and associated memory systems.An unmanned ground vehicle is usually tasked with locating,interrogating, and disrupting dangerous devices. These devices mayinclude improvised explosive devices, unexploded ordnance, or anythingelse a human operator would prefer to be far away from. Unmanned aerialvehicles may surveil their environment, discharge conventional weapons,and conduct electronic warfare. According to the present disclosure,both types of unmanned systems (i.e., ground, surface, submersibleand/or aerial) may benefit from self-contained payloads communicatingdirectly to the OCU through the infrastructure provided by the platform.

FIG. 1A is an overview of a robotic system. The system generallyincludes two modules: a control module 10, and a vehicle module 20.Control module 10 includes operator control unit 12, which is theprimary system the operator uses to teleoperate the robotic vehicle andattached payloads. A radio 14 may be coupled to, and in communicationwith the OCU. Radio 14 may be located internally to laptop OCU orseparate as an external payload. An optional controller 16, such as agamepad (e.g, video game controller) may provide the user an intuitiveinterface to move and articulate the platform and payloads as well ason-screen OCU navigation. The vehicle module 20 may include a robotplatform 22, which serves as the hub to distribute power and/or databetween payloads. Platform 22 may optionally support mobility componentssuch as tracks, wheels, and the like. Platform 22 may be coupleable to anumber of payloads. In FIG. 1A, platform 22 is coupled to a manipulatorpayload 24 consisting of an articulating arm and end effector (gripper)to allow the operator to remotely interact with the environment. Acamera payload 26 provides visual feedback to the user. Optionally,camera payload 26 may articulate and provide illumination to increasethe field of view and objects at a distance or in low lightenvironments. A robot radio payload 28 may provide communication betweenthe platform and the OCUs. Robot radio may use 802.11 WiFi and maymanage an ad-hoc or access point network. Payloads may be coupled to theplatform with a common interface between the platform and payloads toprovide power, ethernet, CAN bus, and/or EIA-232 serial portconnectivity.

As shown in FIG. 1B, a single OCU 12 may communicate via radio withmultiple platforms. In this examples, platform A and platform B havedifferent payloads. Additionally, the same controller may be used tocontrol both platforms. According to the present disclosure, the payloadprovides integration with the rest of the platform and OCU and is readyfor use by the operator. The present disclosure achieves this withoutrequiring that the OCU know about the payload ahead of time. That is, noupdate to the OCU is necessary prior to equipping the system with a newpayload. Instead, the system is designed so that the payload teaches theOCU how to talk to it. When a payload is plugged in to the platform andthe system powered on, the payload will be able to communicate to thesystem through an Ethernet bus. Messages compliant with the Robotics andAutonomous Systems-Ground Interoperability Profile (RAS-GIOP) standardswill allow the components to discover each other. A special JAUS servicecompliant to AS5684 (https://www.sae.org/standards.content/as5684/) wasdeveloped so the OCU can interrogate the payload and request its systemdefinition. The definition message may provide a link to a series offiles accessible over Hyper Text Transfer Protocol (HTTP) or HypertextTransfer Protocol Secure (HTTPS) that further define the payloadcomponents for the OCU. HTTP/HTTPS is used as is supports easy transferof files using the benefits of Transmission Control Protocol (TCP) anddue to its ubiquity is easy to support on almost any payload. Thesecomponents include a link to user interface elements, hand controllerrequests, and all elements needed to fully integrate any payload into anunmanned system.

FIGS. 10A and 10B show the OCU screen before and after a payload isplugged in. The definition files provided by the payload are interpretedrather than executed. Using an interpreted language rather than acompiled language, the payload definition files can be processed by anumber of OCUs and other payloads. This alleviates the need to target aparticular CPU architecture or operating system that the payload'sdefinition file must be compatible with. Therefore, the same definitionfile can integrate, on-the-fly, with the system being implemented onWindows, Linux, iOS, MacOS, and Android OCUs and payloads using CPUarchitectures such as Intel or ARM.

In some examples, the system may support multiple payloads and willdynamically reconfigure the interface to give the operator theexperience that the total set of payloads was designed as one completesystem. This technology allows capability payload designers theflexibility to mix and match, improve, and upgrade capabilities withoutneeding to develop and field a new configuration of the software. Themessages provided by the payload are interpreted at runtime and do nothave to be saved on the OCU or other payloads. This allows the OCU andother payloads to maintain their software, hardware, and firmwareconfiguration and avoids the added cost and complexity of becoming aunique item in logistics systems, such as those of the DoD. The OCUconfiguration never changes which means an OCU may be used on multiplesystems without requiring any update or configuration change. On powerup, an OCU that connects to an unmanned ground vehicle may be identicalto an OCU that connects to an unmanned aerial vehicle. It is up to theplatform configuration and connected payloads to define the rest. Theplatform configuration is actually treated the same way as any otherpayload. This configuration allows controls such as driving or flying tobe treated the same way as rotating a camera.

Hand controllers are one of the main interfaces between the operator andthe OCU. However, a payload may be connected to different platforms thatuse joysticks, gamepads, and touch controls. The present disclosuresolves this by abstracting the payload's needs into requests that arefulfilled by the particular OCU software. A payload may request ananalog joystick to control steering and the OCU fulfills that requestthen shows the operator a diagram of how the features are mapped tophysical actions. This allows the OCU to be realized in different formfactors while supporting the same platform payloads.

Aside from providing basic user interface elements such as text fields,buttons, drop down menus, and other indicators, a powerful scriptingengine is used to give the payload a great deal of flexibility in howthe OCU interacts with it. These interactions, messages over the radiosbetween the OCU and the payload through the platform, must not bepropriety but utilize established messaging standards. During thepayload interrogation process, the payload provides the standard JSIDLmessage definition in XML defined by AS5684 to the OCU so the system canlearn how to talk to it. This feature allows the OCU to learn new JAUSmessages based on the definition provided by the payload.

In some examples, the OCU may display a three-dimensional model of theplatform and any attached payloads. This model will build itself as morepayloads attach and change its shape depending on messages from thepayloads. For a ground vehicle with articulators, this may show anarticulator rotating around an axis as the robot moves around. Foraerial vehicles, this may be represented by rotary wings or changing theorientation as the platform moves. In either case, the platform designerknows what their platform looks like and the payload designer knowstheir payload. The 3D models contained in each will be aggregated by theOCU to provide the operator a complete picture of the unmanned system asit interacts with its surroundings.

The system may further include security features, such as hardwarefeatures to impose security on the platform side. In one example, when apayload is connected, it receives power but not an Ethernet connection.A cryptographic exchange is performed by the payload to the platformover the EIA/TIA-232 standard using a symmetric key challenge andresponse along with an asymmetric key signature. Once the payload hasproperly identified itself, the Ethernet port is unlocked and availableto the payload. Only payloads authorized by a program manager or otherend customer may gain trust of the system. To the operator this occursin the background in a fraction of a second and in no substantial wayimpacts setup time. In addition to these security features, all scriptsdefined by the payload and interpreted by the system operate in asandbox. This sandbox environment prevents unauthorized access to therest of the OCU system and one payload affecting another.

FIG. 11 shows an unmanned platform that implement the system and methodsdescribed herein. The platform is a lightweight (e.g., less than 20pounds) chassis with four rotating tracked flippers. Small ground robotsusually suffer from limited mobility and the four rotating tracks allowthe robot to drive through rough outdoor and urban terrain. The uppersurface reveals four payload mounting connections complete with fourpayload connectors. Each payload connector provides power, Ethernet, anda connection to the Aurelius Unlock system. Each payload port andmounting position is the same so the operator can rearrange payloads asdesired. According to the present disclosure, even well after the systemis fielded, the operator may add and remove payloads (e.g., an arm,gripper, camera, radio, or other sensor) without ever having to updatethe system.

Initially, the payload must identify itself with the robot chassis inorder to gain connectivity identify its physical location on the chassis(payload port A versus payload port C). After that it has ethernetconnectivity and can receive broadcast message from the OCU, chassis,and other payloads. Once a broadcast message is received, a response isgenerated which starts a chain of events for the OCU to configure itselfto talk to the payloads of the robotic system. This includes the payloadsending a message that reports the services it provides, along with anHTTP/HTTPS URL to a definition file in the Qt Modeling Language (QML)format that is interpreted by the OCU and instantiates multiplecomponents for the user interface. The QML file also provides HTTP/HTTPSlinks to XML files conforming to AS5684 that provides the definition ofthe message interface specification of the previously defined servicesthey payload implements. This means a set of messages that the OCU hasnever been programed to communicate with are provided by the payload andinterpreted by the OCU at a high level so it can talk to the payload asa low level confirming to the AS5684 standard.

Other implementations of the AS5684 standard use software codegenerators that send messages in a particular format but are notdynamic, meaning they would need to be reprogrammed and reconfigured forevery new payload. According to the present disclosure, the systeminterprets the XML file describing the definition according to thestandard and is able to send and receive messages without the OCUneeding to be reprogrammed or reconfigured. This is done by loading theXML file into a tree structure and then generating outgoing or parsingincoming messages based on that tree structure.

Once the underlying communication is established, the OCU may map thefunctions of the payload to the human-machine interface components suchas keyboard and game pad. In its current implementation, an Xbox handcontroller is used. There are several hard-coded buttons that the OCUuses for navigation but the remaining buttons and axes are madeavailable to the different payloads. In the QML file are generallydefined requirements the payload has for the OCU. For example, drivingrequires a forward ax, reverse ax, and a directional ax. The OCUsoftware maps these general requirements to specific variable axcontrols like sticks and triggers. This provides flexibility if adifferent controller were used with the same payload. The payload wouldbe transferable to different chassis or OCUs without the need forreconfiguration. This is performed by matching the requirements of thepayload to the capabilities provided by the controller. An on-screenimage showing which functions are mapped to which buttons and axes areshown to the operator so they know which buttons to press (FIG. 12A).

Enhanced capabilities like including a special function like “Shift”allow the payload to load more requirements than may be physicalachievable by a single controller. If the user presses the shift button,the on-screen diagram and underlying control mapping will shift to analternate view and control mapping.

After the messaging and control specifications are handled, the QML filemay further define user interface elements such as video windows, maps,buttons, toggle switches, sliders, and even custom components unique tothe payload. These elements can be used by a touch screen, mouse, orthrough Xbox controller navigation (FIG. 12B).

The details of the interaction between the various components of thesystem will now be described in greater detail.

FIG. 2 illustrates one example of payload-platform initialization. Firstan operator powers on the platform which provides power to the connectedpayloads (step 201). The platform operating system and software alsostart. The payload's operating system also starts and launches a primaryapplication and any required background services. Each payloadperiodically sends an identification messages through the serial bus tothe platform identifying itself and continues until a response isreceived (step 202). The identification message may contain thepayload's make, model, version, serial number or other identifyinginformation. Optionally, this message may include a cryptographicsignature of the identifying information using a key provided by a thirdparty. The platform decodes and responds to message and in the replycontains the identifier of the physical port to which the payload isattached (step 203). The response message may contain the platform'smake, model, version, serial number or other identifying information.Optionally, this message may include a cryptographic signature of theidentifying information using a key provided by a third party. Thepayload then sends a message to request access to one or more ethernetbuses supplied by the platform (step 204). This message may be guardedby the platform's identifying information and digital signature. Inresponse, the platform enables one or more ethernet busses (step 205).This action may be guarded by the payload's identifying information anddigital signature. The payload waits for further incoming messages suchas the discovery message from other payloads or the OCU (step 206).

FIG. 3 illustrates one example of payload discovery and initialization.First, the operator powers on the system and operating system loads andlaunches primary OCU application and any necessary background servicesused to interact with payloads (step 301). Periodic “discovery”broadcast messages are sent to all connected payloads (step 302). Thewired and wireless network will ensure the message is routed to alldevices on the ethernet bus. The payload responds to the “discovery”message with a “report identification” message containing the name ofthe payload and a logical address, as defined in SAE AS5710, used forfuture message routing (step 303). The OCU sends a “query services”message to ask the payload to identify and name and version of theservices the payload implements (step 304). In response, each payloadreplies with a list of services and their corresponding version number(step 305). The OCU will send a “Query Payload Definition” messagedirectly to the payload using the logical address previously defined(step 306). The “Query Payload Definition” message may includecapabilities and limitations of the OCU such as screen size, touchscreen capabilities, hand controller capabilities, and/or keyboardcapabilities. The payload responds with a “Report Payload Definition”message containing an HTTP/HTTPS link containing additional payloaddefinitions (step 307). These definitions are generated at runtime basedon the capabilities and limitations provided by the OCU. For example, asmall screen OCU present on a phone may not be provided the same layoutas desired for a large OCU such as a laptop. The capabilities andlimitations provided for the OCU hand controller may also further refinethe payload definition by incorporating multiple controls into a“up-down-left-right” directional pad or to split the controls intomultiple “up-down” controls. This allows for the payload to tailoritself to a wide variety of OCU and hand controller interfaces. The OCUmakes a request for message or response at the HTTP/HTTPS link (step308). The payload responds with file contents over HTTP/HTTPS (step309). The format of the message is in a plain text format such as QML orXML that can be read by a human or computer without needing to becompiled. The physical location ID provided by the platform is includedin this message so the OCU can determine where the payload is physicallylocated with respect to the platform. In some examples, the QML or XMLdefinition file may include additional links to files in the SAE AS5684format providing additional custom message definitions. If these arepresent, the OCU will send an HTTP/HTTPS request for these files (step310). Each payload replies with the contents of the custom messagedefinitions over HTTP/HTTPS (step 311). The OCU parses and validates theAS5684 formatted responses (step 312). Additionally, the OCU will createcustom message objects for any supplied custom messages (step 313).These are used later to encode and decode custom messages they OCUotherwise has not been programed to understand. Any graphical userinterface elements such as on-screen buttons, sliders, toggles, progressindicators, maps, video displays, images, and text the payload requiresthe OCU display to the operator to use the payload are created (step314). Optionally, this may include tutorials or instructions for theoperator so they may understand how to use a new payload. The OCU willalso map any gamepad buttons or joystick controls to requests made bythe payload defined in the QML or XML file (step 315). These requestsare fulfilled as available and the payload may optionally definealternate requests or mappings based on what is available. Anypayload-defined functions provided in a non-compiled language such asJavaScript are executed on the OCU so allow the payload to furtherconfigure the view on the OCU (step 316). Following initialidentification, normal operation may begin. The OCU may send any payloaddefined messages in a bit patterned defined in AS5684 to the payload(step 317). Under normal operation, the platform may respond to thesemessages by changing its physical state or providing data from includedsensors (step 318).

FIG. 4 illustrates one example of a custom message object. Custommessage objects are used to encode and decode AS5684 formatted messagesat a higher level making it convenient for payload developers toconstruct and respond to messages between the OCU and payload.Traditional robotic systems require the OCU to have its softwaremodified to utilize custom messages but by using custom message objectsdefined by the payload, the OCU does not need to be modified to talk toa new payload. The AS5684 defined XML files are parsed and validatedbased on rules defined in AS5684 (step 401). The custom defined datatypes such as numbers, strings, lists, arrays, enumerations, andmodifiers to these types such as “optional”, “required”, or “mutuallyexclusive” as defined in AS5684 comprise a custom message object (step402). A data model is constructed with these data types in ahierarchical format. This data model allows for conversion from multipleformats such as the bit pattern sent on the wire to the payload, a C++interface, and a JavaScript interface so that the payload definedfunctions executed on the OCU can construct properly formatted messagesthat are processed by the payload in AS5684 format and decode responsesfrom the payload in AS5684 format (step 403).

FIG. 5 illustrates one example of custom message object encoding anddecoding. First, a payload defined JavaScript function is called by anevent from either the operator or the OCU's payload managementcontroller. This function creates a custom message object (step 501).Some messages have no arguments, optional argument, required arguments,or a mix of optional and required arguments. For any required argumentnecessary to make a message valid, the custom message object model willpopulate them with default values as defined (step 502). JavaScriptfunctions that are provided by the payload and executed on the OCU canmodify these arguments. For example, the operator pushes on the joystickcausing an event to call a function which creates a message that tellsthe payload to rotate the camera counterclockwise. Rotatingcounterclockwise is one of the arguments required for the message. Thedestination address is added to the message header so it will be routedto the correct payload (step 503). The data model of the message buildsthe specific bit pattern defined in the XML definition to be sent to thepayload (step 504). The OCU sends the message to the correct payload bymapping the AS5710 destination address to the payload's IP address (step505). The payload may optionally respond to this message (step 506). Thepayload may also send messages periodically such as a heartbeat orsensor data like a GPS location. When the OCU receives the message fromthe payload, the bit pattern is decoded using the data model for thecustom message object (step 507). Using the custom message object, adictionary of arguments and values is generated that can be used for theC++ or JavaScript interface (step 508). A payload defined functionexecuted on the OCU software may subscribed to incoming message eventsand process the message based on the dictionary of arguments and values(step 509). Since the message definition and corresponding functionsprocessing the messages from the payload are all defined by the payload,the OCU is able to interact with a payload without any prior knowledgeof how it operates and the discovery and initialization procedures teachthe OCU how to talk to the payload.

FIG. 6 illustrates one example of payload-defined human-machineinterface. A payload may connect to different platforms and differentOCUs and must not expect a specific human-machine interface (HMI).Rather than map functions to specific buttons, the HMI manager mapspayload defined requests to real buttons or real joysticks. The payloaddefinition in QML or XML defines a set of requests and correspondingcallback functions that are executed on the OCU (step 601). The HMImanager builds a model of available controls based on the physicalcontroller connected to the OCU (step 602). Optionally, default valuesfor navigation, accept, back, and cancel may be reserved for on-screencontrols and not made available to payloads (step 603). Requestedcontrols are mapped to physical buttons based on the type of button ornumber of axis requested (step 604). For example, steering requires aleft and right axe but acceleration requires a single axe. The HMImanager builds a table of payload events to payload defined callbackfunctions that are executed on the OCU (step 605). The HMI managerbuilds an image for each payload's mapped controls to be displayed tothe operator so they are aware of what button maps to which function(step 606). This is generated dynamically and may be adapted dependingon the physical controller connected to the OCU. During normaloperation, the operator presses on a joystick (step 607) to, forexample, accelerate the platform forward. The HMI manager processes thejoystick event and matches it to the corresponding payload definedcallback function that is executed on the OCU (step 608). The callbackfunction may generate a message using a custom message object that willbe sent to the payload (step 609). In the above example, to accelerateat a rate corresponding to where the user moved the joystick.

FIG. 7 illustrates one example of a payload-defined on-screen controls.On-screen controls are populated when the payload is discovered andinitialized (step 701). The OCU may utilize a top navigation bar and abottom status bar with icons and text labels. Each payload must providea navigation icon in the QML or XML payload definition. A payload mayoptionally provide status icons that can be changed during operation(step 702). For example, a payload may provide a compass value and theicon will rotate as the robotic platform rotates in its environmentgiving a visual indicator to the operator. Additionally, the payload mayoptionally define user interface elements in the QML or XML payloaddefinition used for controlling components on the payload (step 703).These may consist of buttons, menus, toggle buttons, icons, radiobuttons, check boxes, or a custom widget defined by the payload. Forexample, a camera payload may have a button to control an illuminator.The operator can toggle this when they encounter a low-light situation.The widget defined by the payload is mapped to a callback function thatwill be executed on the OCU (step 704). Under normal operation, when auser wishes to interact with a payload, they will navigate using thejoystick, click the widget with a mouse or touchpad, or tap the screenif a touchscreen is available (step 705). The widget receives the HMIevent and generates a corresponding action event that is routed to themapped payload defined callback function (step 706). The payload definedcallback function is executed on the OCU and may optionally send amessage using a custom message object (step 707). In the above example,a user navigates to a button using the joystick, presses the standard“Accept” button, the OCU executes the payload defined function whichcreates a custom message object with an argument enabling theilluminator. The payload will decode the message generated using thecustom message object and perform the desired action (step 708). In theabove example, the payload may power on the illuminator.

FIG. 8 illustrates one example of OCU-defined capabilities andlimitations. The OCU software may run on a variety of platforms andhardware such as laptop, desktop, server, tablet, and phone with avariety of human interfaces such as keyboards, mouse, joystick, touchscreen, etc. The OCU software will query the hardware available andgenerate a list of capabilities and limitations for payloads to use toadapt their definitions to the platform to which they are connecting(step 801). After discovering a payload, the OCU may send thisdefinition along with the request for the payload's definition file(step 802). The payload may parse the capabilities and limitations ofthe OCU (step 803). The payload may adapt and match the control mappingsit requires for operation to what is available on the OCU (step 804).Given the OCU screen size provided by the OCU as part of thecapabilities and limitations, the payload will resize any visualcontrols so they may fit on the screen (step 805). In addition to size,styles and color pallets provided by the OCU may be adapted by thepayload. This allows for different payloads to offer the same userexperience on a given OCU. Based on the available controls such asjoystick axes and buttons, the payload will generate the HMI requestsfor the OCU to fulfill (step 806). By taking into account what isphysically available on the OCU, these requests from the payload aremore likely to be fulfilled by the OCU. Protocol definitions aregenerated given the available HMI interface (step 807). These allow theOCU to communicate to the payload without any prior configuration. Thepayload generates a complete definition describing how it must beintegrated into the OCU by the OCU (step 808). This may include apre-defined 3D model of the payload and the location identifier thepayload received from the platform to which it is connected. If thepayload represents the base chassis, the definition will specify it isthe origin of the platform. If the payload contains connection pointsfor other payloads (as in the case of the chassis), it will providelocation identifiers that connected payloads will include in theirdefinition. The payload may transmit this dynamic file to the OCU (step809). The OCU parses and validates the definition provided by thepayload (step 810). Graphical elements such as buttons, toggles, menus,images, etc. are generated by the OCU using the payload definition as aguide (step 811). The mapped functions on any attached controller arecreated and a graphical representation is generated to provide theoperator an understanding of the functions of the buttons and axes onthe controller (step 812). Given the 3D model and connection identifierprovided by the payload, the OCU will generate a visual representationbased on the origin mapped from the base payload (chassis) (step 813).In the case of stacked payloads such as a camera connected to an arm,the origin point of the camera will be the arm and the OCU will move thecamera and arm together as the arm moves.

In addition to communicating with the OCU, the payloads may alsocommunicate with one another. FIG. 9 is one example ofpayload-to-payload communication. Payloads may communicate theircapabilities, limitations, requirements, and physical envelope to otherpayloads requesting such information. All components such as thechassis, payloads connected to the chassis such as an arm, and payloadsconnected to other payloads like a camera to an arm, use the sameprotocol. Payload-to-payload communication is a desirable component inbuilding a dynamic robotic system that must incorporate and integrate awide range of payloads. Simply loading a configuration file on the OCUis not sufficient when deploying real-world robotic systems to endusers. For example, a robotic platform may contain multiple arms orarticulating cameras that must avoid interfering with each other. If anarm were to collide with another arm while performing a dangerousoperation, the robotic system and user may be at risk. To avoid theseconflicts, the payloads may discover each other and coordinate theiractivities with each other to avoid damage. While the operator mayattempt to avoid collisions between, for example, and arm and a camera,delays between the OCU and robotic platform require a dedicated solutionon the part of the payloads that move within their environment. This maybe accomplished by a single payload periodically sending messages of itsphysical envelope to other payloads that request updates. If two armsare attached to a robotic platform, they will subscribe to each other'schanges in physical envelope and guard against motion that may cause acollision. When a payload determines its motion must be stopped to avoidcollision, it may send a message to the OCU to indicate to the operatorthat the collision was avoided. Shifting the role of collision avoidanceto the payloads, rather than the platform as in other implementations,the robotic system can better support a wide range of automaticallyintegrated payloads. On startup, payloads will broadcast discoverymessages for others to respond to (step 901). Upon receiving a discoverymessage, a payload will respond with its own capabilities, limitations,requirements, and physical envelope (step 902). For example, Payload #1will generate a model of all other payload definitions received (step903). This may be updated during operation based on periodic intervalsor based on change events. The payload receiving this message may adaptits operation based on the other payloads' capabilities, limitations,requirements, and physical envelope (step 904). This includes collisionavoidance with other payloads (such as an arm colliding with otherpayloads). If any of the capabilities, limitations, requirements, andphysical envelope change over time, the payload will send an updateddefinition (step 905).

In addition to communication between attached payloads, the roboticplatform may also warn payloads to avoid a physical configuration thatmay jeopardize the stability of the platform to avoid the robotic systemfrom tipping over. For example, an arm may have a long reach and maypick up heavy objects. The arm knows the weight of its individual jointsand can calculate the weight of the item it picks up based on the torquerequired by individual joints. From this information, the arm maycalculate a center of mass and make this available to other payloads andthe robotic platform. If the platform calculates, based on the othercenters of mass provided by other payloads and its own center of mass,that a tipping situation is imminent, the platform may either warn thepayload that is causing the problem, in this case the arm holding aheavy object, or the platform may adapt itself by extending stabilizingarticulators. In either solution, the robotic system is resolving thetipping situation through communication between the attached payloadswithout direct operator intervention.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio player, a Global Positioning System (GPS)receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described is this specification, or anycombination of one or more such back end, middleware, or front endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

It will be appreciated that the various dependent claims and thefeatures set forth therein can be combined in different ways thanpresented in the initial claims. It will also be appreciated that thefeatures described in connection with individual embodiments may beshared with others of the described embodiments.

What is claimed is:
 1. A method of communication between at least onepayload and a platform, comprising: physically coupling the at least onepayload to a platform so that a surface of the at least one payload isin contact with a surface of the platform; providing power to the atleast one payload via the platform; periodically sending anidentification message from the at least one payload to the physicallycoupled platform via a serial bus; decoding the identification messagewith the platform and sending a response message to the at least onepayload; requesting access to one or more ethernet busses supplied bythe platform; enabling the one or more ethernet busses only afterconfirmation of a proper identification message; sending a Qt ModelingLanguage (QML) file from the at least one payload to an operator controlunit, the QML file defining user and logical interface requirements;sending an Extensible Markup Language (XML) service definition from theat least one payload to the operator control unit; parsing the XMLservice definitions; and generating Human Machine Interface (HMI)mapping and button drawings for each of the at least one payload.
 2. Themethod of claim 1, wherein the identification message includes the atleast one payload's make, model, version, serial number and otheridentifying information.
 3. The method of claim 1, wherein theidentification message includes a cryptographic signature.
 4. The methodof claim 1, wherein the response message includes the platform's make,model, version, serial number or other identifying information.
 5. Themethod of claim 1, further comprising the step of creating at least onegraphical user interface element.
 6. The method of claim 1, wherein theat least one payload includes multiple payloads comprising at least acamera and a manipulator.
 7. The method of claim 1, further comprisingsending payload port identifying information from the platform.
 8. Themethod of claim 1, wherein the at least one payload comprises at leastone of an arm, a gripper, a camera, a radio and a sensor.
 9. The methodof claim 1, further comprising carrying the least one payload on theplatform.
 10. The method of claim 1, further comprising transporting theleast one payload via the platform.